describe and visualize of US weather event data

city selection - univariate for time variables

purpose of notebook

(/) describe & visualize single variables (univariate) (-) gather interesting observations for further investigation (-) gather possible new features for extraction

todos: (-) …

information
https://www.kaggle.com/sobhanmoosavi/us-weather-events https://smoosavi.org/datasets/lstw
Moosavi, Sobhan, Mohammad Hossein Samavatian, Arnab Nandi, Srinivasan Parthasarathy, and Rajiv Ramnath. “Short and Long-term Pattern Discovery Over Large-Scale Geo-Spatiotemporal Data.” In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, 2019.
Weather event is a spatiotemporal entity, where such an entity is associated with location and time. Following is the description of available weather event types:
Severe-Cold: The case of having extremely low temperature, with temperature below -23.7 degrees of Celsius. Fog: The case where there is low visibility condition as a result of fog or haze. Hail: The case of having solid precipitation including ice pellets and hail. Rain: The case of having rain, ranging from light to heavy. Snow: The case of having snow, ranging from light to heavy. Storm: The extremely windy condition, where the wind speed is at least 60 km/h. Other Precipitation: Any other type of precipitation which cannot be assigned to previously described event types.
The weather data is provided in terms of a CSV file with the following attributes: Attribute Description Nullable 1 EventId This is the identifier of a record No 2 Type The type of an event; examples are rain and snow. No 3 Severity The severity of an event, wherever applicable. Yes 4 StartTime (UTC) The start time of an event in UTC time zone. No 5 EndTime (UTC) The end time of an event in UTC time zone. No 6 TimeZone The US-based timezone based on the location of an event No (eastern, central, mountain, and pacific). 7 LocationLat The latitude in GPS coordinate. Yes 8 LocationLng The longitude in GPS coordinate. Yes 9 AirportCode The airport station that a weather event is reported from. Yes 10 City The city in address record. Yes 11 County The county in address record. Yes 12 State The state in address record. Yes 13 ZipCode The zipcode in address record. Yes

insights

1. Seattle: events come in waves within the year (peak in Nov, low in Jul) but stable across years,
2. Houston: events seem stable, spontaneous less in Jul and Nov,
3. York: stable but jump to more in 2018, stable across week and year
4. Los Angeles: kind of stable but spike around 2017 and increase 2019 and 2020, over week spikes on monday and thursday, little wave with peak in Jan and low an Jun, distribution over hours shows significant peak at 15:00
5. all: distribution over month days always low at first and last two, distribution over minutes have (different, but all at end of hour) enormous peaks for each city
6. since human judgment seems to be involved, the duration has peaks at the full hours, you even can see the dips in the poisson like distribution
7. duration stats: Seattle median 60 min, q3 120 min; New York median 47 min q3 72 min; Los Angeles median 60 min q3 120 min, Houston median 60 min q3 98 min; Los Angeles and Seattle seem similar in distribution

stories

(!) compare 4 big cities which are all close to the coast but vary in significantly in longitude and latitude

load packages

overview

head(weather)

summary(weather)

   EventId                     Type          Severity      StartTime                      EndTime                          TimeZone     LocationLat     LocationLng     
 Length:12226       Cold         : 365   Heavy   : 439   Min.   :2016-01-01 08:47:00   Min.   :2016-01-01 09:47:00   US/Central:2900   Min.   :29.98   Min.   :-122.31  
 Class :character   Fog          :2920   Light   :6792   1st Qu.:2017-04-08 08:59:45   1st Qu.:2017-04-08 11:06:30   US/Eastern:3656   1st Qu.:34.02   1st Qu.:-122.31  
 Mode  :character   Hail         :   6   Moderate:3428   Median :2018-09-27 02:42:30   Median :2018-09-27 03:02:30   US/Pacific:5670   Median :40.78   Median : -95.36  
                    Precipitation: 139   Other   :   6   Mean   :2018-08-09 22:18:52   Mean   :2018-08-09 23:49:14                     Mean   :38.70   Mean   :-100.68  
                    Rain         :8296   Severe  :1422   3rd Qu.:2019-11-24 10:37:30   3rd Qu.:2019-11-24 12:37:30                     3rd Qu.:47.44   3rd Qu.: -73.97  
                    Snow         : 494   UNK     : 139   Max.   :2020-12-31 23:22:00   Max.   :2020-12-31 23:53:00                     Max.   :47.44   Max.   : -73.97  
                    Storm        :   6                                                                                                                                  
          City        Duration       
 Houston    :2900   Length:12226     
 Los Angeles:2378   Class :difftime  
 New York   :3656   Mode  :numeric   
 Seattle    :3292                    
                                     
                                     
                                     

univariate StartTime

StartTime and EndTime look almost the same, thus we only look at StartTime for univariate exploration Seattle: events come in waves within the year (peak in Nov, low in Jul) but stable across years, Houston: events seem stable, spontaneous less in Jul and Nov, New York: stable but jump to more in 2018, stable across week and year Los Angeles: kind of stable but spike around 2017 and increase 2019 and 2020, over week spikes on monday and thursday, little wave with peak in Jan and low an Jun, distribution over hours shows significant peak at 15:00 all: distribution over month days always low at first and last two, distribution over minutes have (different, but all at end of hour) enormous peaks for each city

startTime_plot <- weather %>%
  ggplot(aes(x = StartTime, y = City, col = fct_infreq(City))) +
    geom_jitter(alpha = 0.2, size = 0.5) +
    theme_minimal() +
    ggtitle("StartTime of the events")

ggplotly(startTime_plot)

# https://r4ds.had.co.nz/dates-and-times.html
# Note that when you use date-times in a numeric context (like in a histogram), 1 means 1 second, so a binwidth of 86400 means one day. For dates, 1 means 1 day.
startTime_freqplot <- weather %>%
  ggplot(aes(x = StartTime, col = City)) +
    geom_freqpoly(bins = 30) + # binwidth = 86400 seconds = 1 day, 600 s = 10 minutes
    theme_minimal() +
    ggtitle("frequency plot for StartTime") +
    facet_grid(rows = vars(City))

ggplotly(startTime_freqplot)

startTime_barplot_wday <- weather %>%
  mutate(wday = wday(StartTime, label = TRUE)) %>%
  ggplot(aes(x = wday, fill = City)) +
    geom_bar() +
    facet_grid(rows = vars(City)) +
    theme_minimal() +
    ggtitle("distribution over week days")
  
ggplotly(startTime_barplot_wday)

startTime_barplot_year <- weather %>%
  mutate(year = year(StartTime)) %>%
  ggplot(aes(x = year, fill = City)) +
    geom_bar() +
    facet_grid(rows = vars(City)) +
    theme_minimal() +
    ggtitle("distribution over years")
  
ggplotly(startTime_barplot_year)

startTime_barplot_month <- weather %>%
  mutate(month = month(StartTime, label = TRUE)) %>%
  ggplot(aes(x = month, fill = City)) +
    geom_bar() +
    facet_grid(rows = vars(City)) +
    theme_minimal() +
    ggtitle("distribution over months")
  
ggplotly(startTime_barplot_month)

startTime_barplot_yday <- weather %>%
  mutate(yday = yday(StartTime)) %>%
  ggplot(aes(x = yday, fill = City)) +
    geom_bar() +
    facet_grid(rows = vars(City)) +
    theme_minimal() +
    ggtitle("distribution over days of year")
  
ggplotly(startTime_barplot_yday)

startTime_barplot_mday <- weather %>%
  mutate(mday = mday(StartTime)) %>%
  ggplot(aes(x = mday, fill = City)) +
    geom_bar() +
    facet_grid(rows = vars(City)) +
    theme_minimal() +
    ggtitle("distribution over days of month")
  
ggplotly(startTime_barplot_mday)

startTime_barplot_hour <- weather %>%
  mutate(hour = hour(StartTime)) %>%
  ggplot(aes(x = hour, fill = City)) +
    geom_bar() +
    facet_grid(rows = vars(City)) +
    theme_minimal() +
    ggtitle("distribution over hours")
  
ggplotly(startTime_barplot_hour)

startTime_barplot_minute <- weather %>%
  mutate(minute = minute(StartTime)) %>%
  ggplot(aes(x = minute, fill = City)) +
    geom_bar() +
    facet_grid(rows = vars(City)) +
    theme_minimal() +
    ggtitle("distribution over minutes")
  
ggplotly(startTime_barplot_minute)

# Setting larger components of a date to a constant is a powerful technique that allows you to explore patterns in the smaller components.
startTime_count <- weather %>%
  mutate(start_hour = update(StartTime, mday = 1)) %>%
  ggplot(aes(x = start_hour, col = City)) +
    geom_freqpoly() +
    facet_grid(rows = vars(City)) +
    theme_minimal() +
    ggtitle("not seeing the use here, but wanted to check out")

ggplotly(startTime_count)

`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

univariate Duration

since human judgment seems to be involved, the duration has peaks at the full hours, you even can see the dips in the poisson like distribution duration stats: Seattle median 60 min, q3 120 min; New York median 47 min q3 72 min; Los Angeles median 60 min q3 120 min, Houston median 60 min q3 98 min; Los Angeles and Seattle seem similar in distribution

duration_histplot_all <- weather %>%
  ggplot(aes(x = Duration, fill = fct_infreq(City))) +
    geom_histogram(binwidth = 1) +
    theme_minimal() +
    ggtitle("Duration of the events")

ggplotly(duration_histplot_all)

Don't know how to automatically pick scale for object of type difftime. Defaulting to continuous.

duration_histplot <- weather %>%
  ggplot(aes(x = Duration, fill = fct_infreq(City))) +
    geom_histogram(binwidth = 30) +
    theme_minimal() +
    ggtitle("Duration of the events") +
    facet_grid(rows = vars(City))

ggplotly(duration_histplot)

Don't know how to automatically pick scale for object of type difftime. Defaulting to continuous.

duration_boxplot <- weather %>%
  ggplot(aes(y = as.numeric(Duration), x = City, col = fct_infreq(City))) +
    geom_boxplot() +
    theme_minimal() +
    coord_flip() +
    ggtitle("Duration of the events")

ggplotly(duration_boxplot)

duration_boxplot <- weather %>%
  ggplot(aes(x = as.numeric(Duration), y = City, col = fct_infreq(City))) +
    geom_jitter(alpha = 0.2, size = 0.5) +
    theme_minimal() +
    ggtitle("Duration of the events")

ggplotly(duration_boxplot)

library(dlookr)

weather %>% 
  mutate(Duration = as.numeric(Duration)) %>%
  plot_normality(Duration)